another tuning tool

[steveo]

it took way too much time to write.

i think it'll be really handy. it's very good at quickly asking questions about your datalogs

it also does some really neat stuff with tables that can come in handy, like comparisons and interpolations.

under the hood it uses bilinear table lookups just like most ECMs do so it's really good for taking a table from one operating system and mangling it into another one with a totally different layout.
anyway try it out and let me know what you think or not
64 bit windows only release for now
beta
might crash
probably some stuff doesn't work

http://ecmhack.com/tablehack/

[bobcratchet555]

looks fun - the ability to get tables from other OS is very convenient. i havent tried it yet but will in the near future. appreciate other enthusiasts taking the time to share their work with the rest of us.

[steveo]

some crash courses:

analyze log (example: VE)
1. press datalog and select log and hope it does not crash
2. press analysis
3. click table layout. choose your x/y data (MAP and RPM probably) and either type or copy and paste some column/row values in there, or press 'auto layout'. they can be any reasonable format, it'll figure it out.
3a. if you had to type them in, probably click SAVE so you can RESTORE it next time. this saves the layout for use across the tool
4. go back to the main screen and 'select data' (long term trim maybe)

done

add filters to suit, should be self explanitory

you can 'save' the entire analysis profile too, including data, filters, etc. and restore it for different datalogs later.

table interpolation (ex: port a VE or timing table from one ECM to another):

1. create a new 'table'
2. set a layout matching your original table
3. paste your data there
4. click layout again and enter your desired layout. it'll see you have data and ask what to do with it.

compare should be obvious, select two tables (not a datalog), but whats not obvious is the compare is dynamic, if you alter the source data the comparison is always up to date. its also totally interpolated so if you compare tables with different row/column values you can do linear interpolation there too.

graphing should be obvious too, right now 2d plot graphs only work for datalogs, 3d graphs for tables


it's really designed for people that would usually copy/paste their data too/from a spreadsheet, so it's the kind of thing you want your ordinary tuning tool open in another window and this just fills in any gaps

[kur4o]

I have been thinking about something like this for a long time.
For linear layout a nice one will be-> set x,y-> initial;final points->step

And another very cool one, extrapolate a table with extra rows,colums.
You have 100,200,300,400 for a row->extrapolate and it adds extra rows with 120,140,160,180,200 and so on and fill the data in.

[steveo]

You have 100,200,300,400 for a row->extrapolate and it adds extra rows with 120,140,160,180,200 and so on and fill the data in.

it does that!

the data that's 'off the edge' is the hardest. right now this tool lets you continue the slope or just set it null and you can fill manually.

i will add a spline interpolation method soon as well, for tables that we know are actually curved too, like a maf table or whatever.

[steveo]

but another thing is it can compare two tables with the same linear interp

so you can directly compare a table that is 100,200,300,400 with 120,140,160,180,200,300,350,380,400 or whatever

this stuff is really common when you tune subarus and stuff thats' why i needed it so badly for myself

[WASyL]

awsome tool, comes handy when doing lots of tuning with different ECM/PCM but similar engine setups.

[steveo]

please test it. one thing i know is broken is 2d table interpolation. if you want to do 2d table interpolation for now just make a 3d table with a single column and it'll work, rather than making a 2d table.

next version:

- make csv parser more robust and faster
- use log timestamp if available
- XDF table layout import (since you can't copy paste that stuff from tunerpro, it takes too long to make them)

[steveo]

http://ecmhack.com/tablehack/

new version:

- way better way faster CSV import. parses, error checks, and builds its table structures pretty quickly, average under 200msec per MB of log data on an i5, so even big 10MB+ logs should load in seconds. as a 64 bit program you could load gigs of logs if you wanted to, i've only tested up to 500mb of test data. added a progress bar for massive csv files so you know it has not crashed. handles backslashes, quotes, and literal quotes (""=") so it should work with any standards compliant csv generator. it still parses things in quotes as numbers if they are numbers, though (actually it stores two copies of your log, one as strings one as converted numbers where strings = 0.00)

- fixed some glitches with the table viewer, scrolling through hundreds of thousands of lines is pretty smooth

- some options for datalog import, optionally continue parsing a CSV with line bugs (columns per line != columns in header, etc), i know some log generators are super broken.

- enabled time axis (for lag filter in analyzer as well as grapher), must be a decimal time to work, will add a timestamp converter later.

- XDF importer. select your xdf and it will list any tables it feels are viable. seems to work well against EEX but please test! it will probably crash with a malformed XDF but who knows.

so now it'll load log(s), load xdf table, select data, and analyze anything vs anything vs anything filtered by anything with arbitrary data lag in a few clicks and a few seconds.

next up are more log viewing improvements like selection data tracing (for example if you select analysis cells of interest it'll highlight or filter your log appropriately) so you can view the parameters of knock events for example. you will then be able to select entries in the log and they will move a cursor in the graph if you have one, so you can 'drill down' into events more easily.

[steveo]

i am working on a new version that will do much more advanced analysis for tables that the ECM does linear table lookups on (think most VE and maf tables)

doing all this linear/bilinear lookup stuff trying to make it 'think' like an ECM got me thinking

lets define an example table that has 20,40,60,80,100 as columns.

the traditional method, lets say we have a data point of 6 with a lookup value of 25, we go okay, the first cell is close to that value, so, add the data point 6 to the first cell's average. in otherwords we do nearest neighbour interpolation of the data only.

the results are good on a large sample set because of a crapload of averaging smoothing the results

but this is not really how the data point would be seen by the ECM for a table like a VE table that the ECM does linear interpolation on.

what we actually are saying when we log a lookup value of 25, is that we have a data point that affects LINE that has both its slope and gain defined by the values of the first two cells of that table

so in effect what we should do is calculate gain and slope of that line by manipulating the adjacent 2 (or 4 in the case of a 3d table) cells for each data point

does that make sense?

[kur4o]

on motorola cpu build in table lookup opcodes. you need to have not only predefined decimal points, but need to have it hex.

the value used for lookup is usually set with min max as 00-ff or for signed ones and 16bit tables 0000-8000 or 0000-ffff than values are converted to account for that and being used in table lookup. if x-row is scaled from 00-ff 00=400; ff=3600 with some divider of lets say 16 points. lookup value is divided by 16 that it finds the row and between 2 adjacent cells make an average between the 2 cells and multiply with the factor that is left from devision. Hope it that makes sense.

That might not be the case with ppc cpu, or some other cpu that uses scalars set for each table that defines the axis points[ so it is not linear].


I hope to get you some decomplied routines used with PPC cpu so you can figure something out of it. Usually all gm 2d and 3d tables have scalar infront of the table that defines the number of x and y points and are used for devision.

The most newer stuff have mostly floating point tables, there may be totally different math.

some examples for linear motorola tables

100-200-300

add points from 100- to 200 it wil be best if you know how many points are in hex. but that may be too hard to guess.

so you get 100~110~120~130~140~150~160~170~180~190~200

value at 100=35 value at 200 = 70

so you find the spread between 70 and 35 in our case it is 35 and multiply with factor 1.1 for 110 , 1.2 for 120 and so on till 190 with *1.9

first draw horizontal axis than extrapolate vertival axis using newly added extrapolated x axis data.

I hope any of that make sense.

[kur4o]

Some clarifications.

when you find spread lets say 20 and 100 spread is 80 you got 10 extrapolated points between

so for point 0[20] it is 80*0+20
point1 is 80*0.1+20
point2 is 80*0.2+20
and so on

[steveo]

totally makes sense, although i think if i do the math in floating point, the results will be just as usable when scaling tables from lower resolution ecms. i'll have to do some tests on how well it works on a real VE table or whatever. when you have two axes (for a 3d table) things get a bit more complicated. https://en.wikipedia.org/wiki/Bilinear_interpolation

this is a great approximation that i've been using with success, the compiler reduces it to a few instructions and it runs really quickly

  double x2x1, y2y1, x2x, y2y, yy1, xx1;
  x2x1 = x2 - x1;
  y2y1 = y2 - y1;
  x2x = x2 - x;
  y2y = y2 - y;
  yy1 = y - y1;
  xx1 = x - x1;
  return 1.0 / (x2x1 * y2y1) * (
        q11 * x2x * y2y +
        q21 * xx1 * y2y +
        q12 * x2x * yy1 +
        q22 * xx1 * yy1
        );

[steveo]

ok i really suck at math and i wish i had paid attention in linear algebra but i THINK figured out how to effectively reverse 3d linear interpolation of a table lookup without having to cover an entire whiteboard in math. it's about four times as computationally intensive as just a lookup as i have to do four transforms, one for each 'encompassing cell' involved in the transform, as from each cell i change the 'viewpoint' of the 3d shape that effectively joins the four encompassing cells involved in the lookup. someone better at math might be able to do it more efficiently later but this seems to work. i will make it less crashy and do more testing. early results are very promising. data scattered mostly between the cells seems to resolve as expected.

[kur4o]

This is decompiled code for 3d table lookup with e38 ecm. PPC cpu.
It doesn`t make sense to me. you can try to figure it out.

FOr sure v4 an v7 is the count of axis points.