BitHoist on Server

[historystamp]

Is there still an interest on running BitHoist on the Gearhead-efi server?

I understand BitHoist to identify a bin file. Given a bin file, BitHoist tells you want it is.

I've looked into this a little more.

Questions:
What Operating System does the server run? Windows?
What web server? Apache?
Have you gotten BitHoist to run on the server from the command line? This would be a necessary step.

What programming language do I use? I'm looking at PHP. It's free.

The general idea:
1) display "welcome to BitHoist" web page. User selects what bin file to analyze.
2) The PHP script that I would write gets invoked on server
3) Bin file is uploaded to server.
4) Program BitHoist gets invoked on server. It runs as if it was invoked from command line.
5) The PHP scripts get the results from BitHoist.
6) Script sends back new web page with the results.

Robert

[EagleMark]

Sure! Why not have another cool online service? But I can't even get it to run on my windows computer...

Sever is Linux, Apache and PHP is not an issue. All librarys are enabled. If you can make it work I can install it in HTML pages like BCCFind and change paths to server...

[1project2many]

Bithoist outputs to stdout which may or may not produce data you can use for the php script. stdout is configurable on the server and could change. A simple code change would ensure output is directed where needed. Does the output need to be formatted to html or can the script handle that?

There are quite a few calibrations that bithoist does not recognize so it may be worthwhile to fully learn how the program works in order to teach it new ones.

[historystamp]

PHP supports an exec command that says run this file in a command line and return the results to a PHP array. All the HTML formating would be done in PHP.

There are not any comments in the code besides the Open Source statement.


-------------------

Here how I compiled and ran bit hoist. Give it another try. If you run into problems, I'll install Linux & compile there.

Look for "mac $". This is what I typed in.

Just the commands.


cd BitHoistServer/
pwd
ls
cp -i Makefile.ix Makefile
pwd
make
# at this point, I want to a window to copy the akal.bin to this folder.
ls
pwd
# akal.bin is last becase it starts with a lower case lettter!
./bithoist -id -mask akal.bin





# put the bithoist stuff in some folder.
# cd to that directory
#
# also, copy the akal.bin to the same folder as bithoist
mac $ cd BitHoistServer/
mac $ pwd
/Users/mac/BitHoistServer
mac $ ls
BitHoist.c Exormacs.c IdentifyMask.r Mostek.c Storage.c
BitHoist.h Exormacs.h IdentifySpec.r Mostek.h Storage.h
BitHoist.r ExtractBCC Intellec.c Options.c TekExtended.c
CRC.c HostTypes.h Intellec.h Options.h TekExtended.h
CRC.h Identify.c License ReadMe TekHex.c
ChangeLog Identify.h MacObject.c Series.c TekHex.h
Chunks.c Identify.r MacObject.h Series.h ToDo
Chunks.h IdentifyBCC.r MakeFile.mac Signetics.c
EditBCC IdentifyBCCTest.r Makefile.ix Signetics.h

# copy the Makefile.ix to a new file named Makefile
mac $ cp -i Makefile.ix Makefile
mac $ pwd
/Users/mac/BitHoistServer
# compile program bithoist
mac $ make
gcc -c BitHoist.c -o BitHoist.o
gcc -c Storage.c -o Storage.o
gcc -c Options.c -o Options.o
gcc -c Mostek.c -o Mostek.o
gcc -c Signetics.c -o Signetics.o
gcc -c TekHex.c -o TekHex.o
gcc -c Intellec.c -o Intellec.o
gcc -c Exormacs.c -o Exormacs.o
gcc -c TekExtended.c -o TekExtended.o
gcc -c Series.c -o Series.o
gcc -c Chunks.c -o Chunks.o
gcc -c CRC.c -o CRC.o
gcc -c Identify.c -o Identify.o
gcc -o BitHoist BitHoist.o Storage.o Options.o Mostek.o Signetics.o TekHex.o Intellec.o Exormacs.o TekExtended.o Series.o Chunks.o CRC.o Identify.o
mac $ # at this point, I want to a window to copy the akal.bin to this folder.
mac $ ls
BitHoist* EditBCC IdentifyMask.r Mostek.h Storage.c
BitHoist.c Exormacs.c IdentifySpec.r Mostek.o Storage.h
BitHoist.h Exormacs.h Intellec.c Options.c Storage.o
BitHoist.o Exormacs.o Intellec.h Options.h TekExtended.c
BitHoist.r ExtractBCC Intellec.o Options.o TekExtended.h
CRC.c HostTypes.h License ReadMe TekExtended.o
CRC.h Identify.c MacObject.c Series.c TekHex.c
CRC.o Identify.h MacObject.h Series.h TekHex.h
ChangeLog Identify.o MakeFile.mac Series.o TekHex.o
Chunks.c Identify.r Makefile Signetics.c ToDo
Chunks.h IdentifyBCC.r Makefile.ix Signetics.h akal.bin
Chunks.o IdentifyBCCTest.r Mostek.c Signetics.o
mac $ pwd
/Users/mac/BitHoistServer
mac $ # akal.bin is last becase it starts with a lower case lettter!


mac $ ./bithoist -id -mask akal.bin
---- cut here ----
0x0, 0x1000, scan, seq, sum, mask, spec, 0x57ba9cf4,
#ifndef rez
1, 1,
#endif
begin
0x0, 0x1000
end,
begin
sumlo, sumhi
end,
---- cut here ----
File 'akal.bin'
Mask ID: Unknown


mac $

[1project2many]

It looks like you've got it working on your machine. You happened to pick a calibration which bithoist is unaware of. Try the file "sunbird.bin" which is included in the zip file I uploaded. You should see


C:\bithoist>bithoist -id sunbird.bin
File 'sunbird.bin'
Address: 0xc000,0xffff
CRC: 0xf3680b8f
Mask ID: 58
ID byte: 0x58
Sum word: 0xd1ee
Sequence: Dec 3, 1990 #1420 (0D2A058C)
Scan ID: 3411
BCC: AUWR3607 (stock)

I only offer to compile a version on Intel platform because I don't know the portability of a file compiled on a Mac.

[historystamp]

Where did you upload Sunbird.bin? I'll need a link.

Robert

[1project2many]

It's zipped up with bithoist in the "uphill to school both ways" thread.
http://www.gearhead-efi.com/Fuel-Injection/attachment.php?attachmentid=220&d=1318993955

[historystamp]

plus I've been reading through bithoist code to understand how it works and maybe we can put an ID program up as well.

I think I will be able to run on this web site from a technical perspective. I'm concentrating on writing a PhP script to invoke the BitHoist program.

I'm holding off trying to figure out how to add more bins to the program.

Robert

[historystamp]

What I'm thinking of doing will require the existing BitHoist program to be run on the server. The question is will GoDaddy let a program such as BitHoist run on their server. In other words, will GoDaddy let someone upload a binary file and let a PHP script run on the GoDaddy server?

I was thinking EagleMark was running a personal web server.

Robert

[EagleMark]

I was thinking EagleMark was running a personal web server.

Robert :mad1: Oh no, not falling for that again! No need nowadays!


What I'm thinking of doing will require the existing BitHoist program to be run on the server. The question is will GoDaddy let a program such as BitHoist run on their server. In other words, will GoDaddy let someone upload a binary file and let a PHP script run on the GoDaddy server?

RobertAll forum and web based software is placed in my FTP folders. Programs just need proper path to all server side includes. Anything you need is there!

[historystamp]

I've made some progress on porting BitHoist to run an the web.

I've gotten BitHoist to work in my web server!

:innocent2:

Positive:
-- got a prototype PHP scripts written
-- got linux version of BitHoist compiled
-- lots of debug info in scripts.-
-- lots of checking on input parameters

Problems:
-- just a prototype for now.

Update: BitHoist is putting all/most output to error and PHP only returns normal output. I've used some Unix kung-fu to get around this.

I sure wish there was a little more documentation on BitHoist.

You should upload all the files to the same directory/folder. I placed all these files in a folder called bithoist and placed the folder with all the other stuff I have on my web server.

I'm running Apache/PHP 5.

Robert

[EagleMark]

Well wouldn't that be cool! :thumbsup:

I'll have to find some time to try this...

[historystamp]

Once the php pages and BitHoist are install on the web server, you can ...

Run bithoist:
1) Point your web browser to query.php
2) Select a bin
3) click on the upload button.

This invokes the server based web page of reply.php.

At present, you will see a lot of debug information. The actual output from BitHoist is surrounded by horizontal lines.

I used the sunbird.bin file and I included it in the zip file. Here is a picture of the BitHoist output. You'll need to scroll down in your web browser to see:
1766

[historystamp]

Enhanced security of php code by following advice in Essential PHP Security by Chris Shiflett.

Split reply.php into two files. Should be able to adjust file reply.php to include template code.

Include test case write up. See file testFilesProcedures/testCases.txt

See earlier post in this thread for changed code.

Robert

[EagleMark]

We do appreciate the work Robert! If it works it's just another cool addition to Gearhead-EFI.com team effort.

If we get some bad weather, hurt my back or get insomnia back I will give it a shot!

Right now I have winded down web work with good weather coming in North Idaho and have many chores to do. My butt hurts from all the web work this winter and I need to get some excercise! This is without pulling LT1 motor from wrecked Roadmaster Wagon, resealing etc... and installing in Chevy Caprice wagon.

With everything going to a new 4G server I'm hoping I can give FTP access to folder and guys could have there own folder to work on projects like this. Since I have many client websites on my server space it is just not practical to give complete acceess to server for individual projects... something I would like to do!

[historystamp]

Keep BitHoist in mind. Winter rolls around sooner than we all expect. It gave me a chance to learn PHP. Easier to learn & implement than I had expected.

I put PHP to work on another project I had been wanting to do for the last several years.


I think you can configure FTP to require a userid & password & limit access to one folder.

I'm not sure sure you can limit access enough in a shared environment. It's more of a gentlemen's agreement not to mess with other stuff. I get the impression that there is some protection, but there are known loop holes.

I'll read over this shared access stuff if I can find some library books on the subject. With the recession, libraries are not buying new books. Ten year old computer books don't do it.

Robert

[EagleMark]

Libraries have been taking a beating ever since the internet. Now with e books and such I think libraries are going to be antique stores... I used to enjoy going to library and reading all day and leave with a stack of books for the week, but the internet has replaced that...

Have FTP on my list to look into! I think the issue with 1project2many was CGI bin needed to be in main folder, but your dealing php so would just need absolute path from that folder correct?

[historystamp]

Announcing Beta version of BitHoist on GearHead-EFI server.

Check it out. Post comments & suggestions.

Free virtual beer for any bug reports.

Choose File -- Pick a bin file on your Personal Computer.
Upload

2268

Here is the results.

2269

I am using BitHoist with the -id option. Here is the complete list of options. Let me know if I need to specify any others. Half of them I do not know what they do or why they even exist.

2270

[EagleMark]

:yikes: It works! :thumbsup:

Does not like LT1 bins, may have something to do with options, we need to get Robert Saar and 1projest2many to look at options!

:happy: :rockon: :jfj:

[RobertISaar]

honestly, i have very little clue as to what a lot of those options mean.

[1project2many]

Most of the job of that software is manipulating calibration files. .bin as an extension indicates a file is stored as a binary file. But there are other data formats which were developed by various manufacturers to use with their hardware and / or software. Motorola, Intel, Tektronixs and others all have proprietary data formats. Bithoist will read a file in one format and write it in another acting as a converter between the formats. Almost all of the data used in the tuning community today is in binary format and there's little need for the average hotrodder to be switching between data formats so it's ok to leave the options -o and -b through -sig disabled.

Offsetting data can be helpful if trying to generate a file for disassembly in which the actual program line addresses match the addresses the program would occupy were it in the ecm. If the program actually starts at $8000 in the ecm, then make the first line $8000 instead of $0000 like it shows up in most hex editors. This might help identify various calibrations from different manufacturers although for most of the gm cals we use there's no need.

The include and exclude addresses can be used for calibrations that are made in segments such as the MEFI, or (as I've been told) the Northstar and OBDII cals. These cals have shorter sections inside which have their own checksums. For these programs it could be useful to specify only the sections of cal that are already identified or are relevant to identifying the cal. Again, at this time almost all the cals we work on do not require this additional functionality.

The -mask option is part of the -id process although I haven't spent a ton of time trying to work out how. This is a probably a diagnostic function to be used when teaching bithoist about new calibrations.

All said and done, IMO for the purposes of identifying a calibration -id is the only option that should be enabled.

[historystamp]

Here is what I've been able to figure out on how to use the command options. Just for the record.



-- typical invocation


mac $ ./BitHoist -id Sunbird.bin
File 'Sunbird.bin'
Address: 0xc000,0xffff
CRC: 0xf3680b8f
Mask ID: 58
ID byte: 0x58
Sum word: 0xd1ee
Sequence: Dec 3, 1990 #1420 (0D2A058C)
Scan ID: 3411
BCC: AUWR3607 (stock)




-- Print Tektronix format of hex to terminal


mac $ ./BitHoist -t -stdout Sunbird.bin
%2A6EE4C0000D530D2A058CD1EE58800A9715F100E8
%2A6CE4C0105FD03AF8A002000D00FF00EB001000AB
... clipped ...




-- Note BitHoist sends "normal" output to stderr.
-- To redirect all to stdout, do:


mac $ ./BitHoist -id Sunbird.bin 2>&1
File 'Sunbird.bin'
Address: 0xc000,0xffff
CRC: 0xf3680b8f
Mask ID: 58
ID byte: 0x58
Sum word: 0xd1ee
Sequence: Dec 3, 1990 #1420 (0D2A058C)
Scan ID: 3411
BCC: AUWR3607 (stock)




-- all to a file


mac $ ./BitHoist -id Sunbird.bin >myout 2>&1
mac $ cat myout
File 'Sunbird.bin'
Address: 0xc000,0xffff
CRC: 0xf3680b8f
Mask ID: 58
ID byte: 0x58
Sum word: 0xd1ee
Sequence: Dec 3, 1990 #1420 (0D2A058C)
Scan ID: 3411
BCC: AUWR3607 (stock)




-- When you run with the -id parameter, you will get the range of
-- addresses of the binary data in the bin file. Use 0x before a
-- number to indicate hex.


mac $ ./BitHoist -t -stdout -r 0xc000,0xc04f Sunbird.bin
%2A6EE4C0000D530D2A058CD1EE58800A9715F100E8
%2A6CE4C0105FD03AF8A002000D00FF00EB001000AB
%2A6864C0200017005B2A30000AE0080C0906000000
%2A6054C0304493CAB862FFFDFD320A14647800C804
%2A6F44C040B00BB8A0404F7C101EA703E9043C90BF
%0781010




-- Run with mask option


mac $ ./BitHoist -mask Sunbird.bin
---- cut here ----
0xc000, 0x10000, 0xc000, 0xc002, 0xc006, 0xc008, 0x80290058, 0xb4bb275c,
#ifndef rez
1, 1,
#endif
begin
0xc000, 0x10000
end,
begin
sumlo, sumhi
end,
---- cut here ----

[historystamp]

Does not like LT1 bins, may have something to do with option

Please post bins.

I'd like to see how they fail, so that I can collect a list of enhancements.

The BitHoist code has provided mostly unfathomable to me. It's written in C without comments. Correction, I did find two comments in the code. I don't count the copyright statements.

I think that it doesn't analyze modified bins per comment by 1project2many. I'm assuming these bins need to be added.

I've found references to a program to add a new bin to "database" of bins. ( The "database" is an .h file. ) I get the impression there was a separate program to make the addition which I do not have. My current thought is to run a new bin through BitHoist. Write a program to reformat the data to the format BitHoist expects to find.

I think I'm taking a summer break though. I'll continue on getting the current implementation squared away on the server.

Robert

[EagleMark]

Maybe Robert would know if it has to do with the Motorola Hex output option? I seem to remember LT1 being motorola or two chips or?

Here's a link to Lt1 bins
http://www.gearhead-efi.com/Fuel-Injection/showthread.php?335-16188051-and-16181333-PCM-Information-EE-EEB

It may just need the bin database updated as the program is probably older then 1994 when LT1 in $EE came out. then this would need 16197427 $0E and $0D bins etc... we have a ton of bins if you can figure out how to get them in database...

[RobertISaar]

94-95 LT1 BINs are actually 2 64KB BINs stacked on top of each other, one for each processor in the PCM.

that's about all i got. i'm not really certain of what it being accomplished here, i'll start reading from the beginning again.

[1project2many]

Based on some of the ID's bithoist has done for me, the mask is identified by looking at the locations of the various table headers and the ID is called out based on scan ID and identified mask. I don't know how to identify headers other than testing for all possible combinations and calculating a checksum or result to match against a database. If a mask is ID'd then the next step is checking scan ID against the database. Finally, the GM checksum is calculated based on the data against the chip and compared to the checksum for the stock calibration in the database.

First test is to run the same .bin through bithoist with altered scan IDs. That would tell us if the ID follows the scan id once the mask is identified. Then you could take a stock .bin and alter table header data (I could help with this) and see what result -id generates.

I'm sure we can figure out how to put more cals in the database.

[RobertISaar]

i actually just did some of that... ran a BFBD file through, info came out correct. changed the mask ID to AA, no results. changed to 8D, no results.

it didn't like my 94-95 3100 flash BINs either. LOL.

[1project2many]

If the mask ID is wrong but the checksum is correct it will (usually) report what the correct mask ID should be. I'm talking about changing the actual table headers, the data which would tell the ecm for example "table begins at $4111 and is 16 cells by 16 cells."

[RobertISaar]

yeah, i read it wrong the first time over.

i just changed the mask ID using the TP hex editor from A1 to AA/8D, so the checksum updated when it was saved.

[EagleMark]

I have a known stock chip (ADSU out of my truck) and it came back as stock. Same chip with HiWay Lean Cruise came back as ? and a few choices so it can and does pick up stock chips!

But another I thought to be stock came back as same chip but did not indicate stock?

So we will need to come up with instructions and guidelines for what it will do and what results mean?

[1project2many]

It will ID some stock cals, and it will ID some masks. What would be more helpful is figuring out how it "learns" calibrations so we can teach it some more.

[historystamp]

Could you post or link to these bins. Thanks.


Same chip with HiWay Lean Cruise came back as ? and a few choices so it can and does pick up stock chips!

But another I thought to be stock came back as same chip but did not indicate stock?

I'm a total novice about these bins. My knowledge is writing the PHP code in interface to BitHoist.


What would be more helpful is figuring out how it "learns" calibrations so we can teach it some more.

I'll look some more at the code. I'm refreshing my knowledge of C by reading "The Joy of C: Programming in C, second edition" by Lawrence H. Miller and Alexander E. Quilici.

My guess is that there was another program that did this. One which we do not have.

I don't mine writing such a program, but it will be a few months before I'd consider starting.

Robert

[1project2many]

> From: "Ludis Langens" <ludis@cruzers.com>
> To: gmecm@diy-efi.org
> Sent: Saturday, December 09, 2000 7:10 AM
> Subject: New version of BitHoist
> > This version knows about 131 different masks, plus 44 mask ROMs (26 C3
> > ECM's worth). For all except six of the masks, BitHoist checks much of
> > the PROM image for a specific bit pattern. If this doesn't match, then
> > the PROM is not considered to be of that mask. BitHoist can also spot
> > 388 "stock" chips.

[1project2many]

I moved this to a separate post to bump the thread.
For multiple files it might be nice to have the option to upload another file immediately following the results.


Based on notes in the readme files, manual edits of the identifyBCC and editBCC files are expected. Looks like Ludis set up the IdentifyBCC.r file with scan ID, ECM no, and BCC of many cals but left the CRC value as "crcUnknown" when he didn't have a known stock chip available to generate one. BAMW is one such example with bithoist generating a crc of 0x7a746f28 but the displayed BCC is "unknown." In this case it's no challenge to replace "crcUnknown" in the .r file with the actual CRC then recompile and test for correct identification. AWWH is correctly identified (same ecm and mask) and the CRC entry for that cal is populated with a valid number. It wouldn't be difficult to check the EditBCC file for similar examples in order to properly maintain the program. And with help from volunteers a list of CRC values generated by Bithoist for known stock calibrations can be made so that updating/recompiling can be kept to a minimum. Note that the CRC does not have to be correct for C3 calibrations to be identified. Those appear to be identified by scan ID only. One major difference which may play a part is the C3 cals contain very little operating code while the P4 and later cals contain all the code and cal data.

And now I'm about to go to bed for the fourth time tonight. If neither of the kids wakes up crying (again) then I just might get some sleep.

[historystamp]

Added the repeat query to the bottom of the Reply page.

I think BitHoist is ready for the slow drive down main street.

http://www.gearhead-efi.com/bithoist/query.php

Robert

[1project2many]

I have confirmed that when the CRC in the identifyBCC.r file matches the generated CRC, Bithoist will report the calibration as stock.

[historystamp]

Was it the idea that BitHoist would contain all the OEM bins? In that way you could tell whether or not your bin was an OEM bin?

-------------

Is the format of the bins documented?

Couldn't BitHost report most of the -id information without the bin being in the .R file?

It should be able to report CRC: 0xf3680b8f for any file. What about the other -id data?

[1project2many]

I'm not sure I understand the questions. Ludis appears to have added data for many of the cals. There are more that can be entered. I can't say what his intention was but I'll bet he expected that more would be added over time.

Bin format documented? We know where the mask ID is located along with the checksum, sequence number, and the scan ID.

Bithoist will report data from several sources. In the case where no data is available within the program it reports "mask unknown." If the mask can be identified and it's from an older C3 ecm BH will report the mask and checksum and suggest calibration ID's (BCC's) based on the scan id. If the mask is from a P4 or later ecm and the calibration is in the list BH will report the BCC and run a CRC check. If the CRC matches the CRC in the list BH will also report the cal as "stock." If the CRC is not in the list or doesn't match then BH will not report "stock." One file which contains this data is identifyBCC.r .

It appears that you're using one file for testing. Here are a couple of examples of what I'm describing:


File '/tmp/phpu8CVHG'
Address: 0x8000,0xffff
CRC: 0x6eb0af54
Mask ID: DAa
ID byte: 0xDA
Sum word: 0xe3dc
Sequence: Jul 18, 1991 #1702 (D7D106A6)
Scan ID: 1641
BCC: AZUP1702

CRC above is not in database or does not match


Processing file BPLH-$0D-4.3-Manual.BIN
( the file name on the server shown below is different ).


File '/tmp/php3azQsB'
Mask ID: Unknown

Cannot find matching data



Processing file BHFM 94 S-Truck 2.2MPFI 5 speed california.BIN
( the file name on the server shown below is different ).


File '/tmp/php8nlHkl'
Address: 0x0,0xffff
CRC: 0x0e773311
Mask ID: 0Da
ID byte: 0x0D
Sum word: 0xcccb
Sequence: None (00000000)
Scan ID: 4941
BCC: Unknown
Found mask ID but no BCC entry for scan ID.


Processing file aixz OD.bin
( the file name on the server shown below is different ).


File '/tmp/phpokB9MJ'
Address: 0x0,0xffff
CRC: 0x5f76b544
Mask ID: 0Da
ID byte: 0x0D
Sum word: 0x9874
Sequence: None (FFFFFFFF)
Scan ID: 265
BCC: BJYA0072

Either file is misnamed or BH only found one entry for scan id



File '/tmp/phpkcOXBp'
Address: 0xc000,0xffff
CRC: 0x83ba62b4
Mask ID: 58
ID byte: 0x58
Sum word: 0xd5dd
Sequence: Feb 4, 1993 #21480 (016153E8)
Scan ID: 831
BCC: BBZA1576 (stock)

Notice "stock" in parenthesis. A matching CRC value for this cal is in the .r file and BH is confident that this file is stock. Over time more CRC data can be collected and the .r file amended so BH will ID more files as "stock" if they truly are.

[historystamp]

Minor clean-ups to BitHoist web page.

Robert

[historystamp]

This link will get you to BitHoist.

http://www.gearhead-efi.com/bithoist


-------------------------------------------

I used Apache Redirect. Place file .htaccess in bithoist directory.

contents. It's one line. Notice partial path on left. Full path on right.:

Redirect /bithoist/index.html http://www.gearhead-efi.com/bithoist/query.php


more details here.

http://www.pa.msu.edu/services/computing/faq/auto-redirect.html

--------------------------------------------

I was able to get a mac program TextWrangler to edit files directly on the ftp server. No having to upload any more.

Robert

[EagleMark]

Robert, we all appreciate your time and things are looking good! :thumbsup:

I'm just swamped with work and don't have time to do my part and get it together with other pages but will do soon!