Split text file into lines

2»

Comments

  • malbrechtmalbrecht Fabric for Houdini Posts: 752 ✭✭✭

    KL is very much like Javascript and almost all other "simple" languages, maybe you just give it a go in straight code instead of nodes?
    I'll have a look later on ...

    Marc


    Marc Albrecht - marc-albrecht.de - does things.

  • seltzdesignseltzdesign Posts: 80
    edited August 2016

    @malbrecht Yeah I tried that, but it didnt really work. I had a look at how you do it in C and it seems straight forward, but not when you have 2 for loops in canvas, which dont work the same as 2 for loops in code.
    Like I said, I never really got the syntax right (errors) and when I did, it either would output nothing or not let me execute the second for loop inside the first one.

    Theoretically it should be something like:

    f = files.size() //number of files
    
    for (i=0, i<f, i++)
    
        s = mat44s.size() //number of mat44's in that file
    
        for (j=0, j<s, j++)
    
            push.mat44(matrixes[i][j]); //matrixes is the 2D array with all mat44's from all files
    

    Right?

  • malbrechtmalbrecht Fabric for Houdini Posts: 752 ✭✭✭

    hmm ... no. :smiley:

    without enclosures each for loop will only execute the next line. So your "for-i" only executes the "s=mat44...", not the following "for-j".
    I haven't looked at your graph yet, so I am not sure about your mat44s, but I guess you want to have some index there as well (can be completely wrong, I'll look-have ASAP).
    The "push" command in the last line actually is a function of an array, but I guess you quoted "virtual code" there anyway ...

    Marc


    Marc Albrecht - marc-albrecht.de - does things.

  • seltzdesignseltzdesign Posts: 80

    Yeah sure, it was just to get across the idea, not actual code. I am willing to learn about enclosures and scopes and those concepts, which have so far eluded me, but I was hoping that Fabric partly negates that need, by letting me do it with visual nodes and blocks. Guess I was wrong about that :/

    There should be some index of course, so after loading the contents of all the files and caching them, I can then recall the contents of each file for each frame on the timeline.

  • malbrechtmalbrecht Fabric for Houdini Posts: 752 ✭✭✭
    edited August 2016

    sorry, I simply meant bracketing like this:

        for(Size i=0;i<max;i++)
            {
            for(Size j=0;j<max;j++)
                {
                do_weird_stuff();
                } // j
            } // i
    

    Gimme another 30-60 minutes :)

    Marc


    Marc Albrecht - marc-albrecht.de - does things.

  • EricTEricT Administrator, Moderator, Fabric Employee Posts: 304 admin

    The ForLoop node isn't plugged in to anything that would trigger it to eval. Get the exec port and plug into the exec port of the main graph and try from there.

    Eric Thivierge
    Kraken Developer
    Kraken Rigging Framework

  • seltzdesignseltzdesign Posts: 80

    @EricT Yes, I am still getting to grips with understanding the exec port and when to plug something into it and when not. So to execute a for loop I need to hook its exec outport into the exec port on the right?

    Whenever I think I am getting close, I am presented with this :/

  • EricTEricT Administrator, Moderator, Fabric Employee Posts: 304 admin

    Can you tell me the exact steps you do to get the crash and let me know which .canvas file I should try this with?

    Eric Thivierge
    Kraken Developer
    Kraken Rigging Framework

  • malbrechtmalbrecht Fabric for Houdini Posts: 752 ✭✭✭

    you need to "read" a graph from right to left, if you want to see what gets executed. Only what has some connection (going through whatever nodes) to the right side of the graph will get executed. If there is no connection, the nodes are just "dead".
    The "exec" port is a "cheap wayout" if you only want nodes to get executed without exposing some result anywhere.


    Marc Albrecht - marc-albrecht.de - does things.

  • malbrechtmalbrecht Fabric for Houdini Posts: 752 ✭✭✭

    Hi,

    not sure if this is what you are after, but at least it does do something ...

    What I did: Connect exec ports to exposure side AND push the mat44[] onto the mat44[][] stack.

    Marc


    Marc Albrecht - marc-albrecht.de - does things.

  • EricTEricT Administrator, Moderator, Fabric Employee Posts: 304 admin

    I took @malbrecht's latest version and changed a few things. However there needs to be more logic and different data structures for storing the frame data.

    First you need to add some logic making sure there is a valid frame file for each frame you change to and add an empty array or something when there isn't.

    Secondly, we are pushing entries into an empty array where its indices start at 0 so if you use the frame lookup it isn't going to work correctly since the first frame is most likely 1001. It'll error out and say there isn't an item at that index.

    Eric Thivierge
    Kraken Developer
    Kraken Rigging Framework

  • malbrechtmalbrecht Fabric for Houdini Posts: 752 ✭✭✭

    hehe ... I was thinking that perhaps I should have tried to understand what's going on, but I really just wanted to give it a go and do something :)

    I think that is one reason for why I struggle so hard with the visual "programming" approach. It lures (me?) into "just get something out and be done" instead of planning, layouting and ... documenting what I am doing, before and while I am doing it.

    A reader function like this is, in my eyes, something that does not work well (in terms of being servicable) with nodes. There are just too many "if" conditions (if !file_exists, if !array, if value>threshold, if ...) which would clobber up the subgraphs. And mixing nodes and KL just for the fun of it (like the graph is built right now, with some split-functions being KL, the loops being nodes etc) gets messy.
    I'll happily write a single KL node that does the complete job - if that is helpful to @seltzdesign ?

    Marc


    Marc Albrecht - marc-albrecht.de - does things.

  • seltzdesignseltzdesign Posts: 80

    @EricT Thanks so much for your file. When I select a folder with the some proper test files (there are 300 of them, each around 1-2 KB), I just get a spinning wheel and Canvas locks up completely - it doesn't quite crash, but as soon as I move the timeline it freezes. And in the log there are a lot of ThrowOutOfRangeException Errors :/

    There aren't really that many tests necessary. The files always start at frame_00000.txt and there is always one for each frame.

    So as it stands the method where I dont cache anything and just read the files straight is the fastest. Now if only there wasn't that problem with the Decompose Mat44 array node. Could you take a look at that please, I started a new thread, but had no replies yet: http://forums.fabricengine.com/discussion/697/bug-decomposemat44array-no-output#latest

    I am attaching the text files again in a .zip if you want to test

    @malbrecht That would be awesome! It doesn't need many checks, just one file for each frame, only frame files in one folder, all have the same naming convention. So just read all the files, push content of each into a 2d array, get one array of Mat44's for each frame of the timeline. Bonus task: turn into XFO's because of the bug mentioned above.

    Thanks so much to both of your for your help!

  • malbrechtmalbrecht Fabric for Houdini Posts: 752 ✭✭✭

    Moin,

    right, I hope I got the task right ...

    Here are two nodes basically doing the same thing: Convert files to Xfos.

    The upper one reads in a single frame file depending on the connected timeline, so you can use this to either act within a running animation OR use it to fill up a variable over the run of the timeline.

    The lower one replaces the timeline by a virtualframe counter, starting with the frame you specify in the node ("start_with_frame"), probably this is value 1. Note that the first Xfo in the resulting array will then be empty - if you want to start the Xfo array with index 0, just adjust the code. The counter stops when there are no more files to read in.

    Both nodes need a filename (string) as input, I added a filebrowser for that. Just select ANY of the ".txt" files in your data input directory, as the whole read-in filename is getting constructed from there. You can name the input files whatever you like, the only expected conventions are: ".txt" ending and a "_xxxx" as last component before the extension point. The number of frame digits is getting read out, so you can have "_xx" as well as "_xxxxxxx".

    I commented the code, hopefully well enough to give you a start. The code does some error checking, but only rudimentary.

    You could do it all in nodes - but what for, it would be a mess, hard to read and have no advantage over a very simple block of code (just my 42 cent, of course).

    The mat44 is filled with a (0,0,0,1) as last Vec4, according to a comment you made further above. If you do not want this, just change the one line that constructs the mat44.

    Let me know if I completely failed ...

    Marc


    Marc Albrecht - marc-albrecht.de - does things.

  • seltzdesignseltzdesign Posts: 80

    @malbrecht thanks so much for your time. I think its close, but not quite there yet.

    I have the non-caching version working now. You can see whats going on there, I can scrub throught the timeline and see the animation that is built from the text files.

    I want it to be the same thing, but rather than Canvas actually reading each file when the timeline is moving, it should read all the files at the start and then just output the relevent array of Xfo's for each frame, putting the whole 2D array of all the arrays of Xfo's into a variable, so its stored.

    Thanks for your effort!

    I will attach my working version, as well as your version inside my working version, so you can test it. You can use the text files from my previous post.

    Best, Armin.

    What it should look like at frame 250:

    What it looks like right now with your file at frame 250:

  • malbrechtmalbrecht Fabric for Houdini Posts: 752 ✭✭✭

    in a discussion right now, just a quick response: The second node reads all data in at once, so the Xfo array contains all data from all files (only the first node reads one file at a time) ... will try to understand your graph a bit later ...


    Marc Albrecht - marc-albrecht.de - does things.

  • malbrechtmalbrecht Fabric for Houdini Posts: 752 ✭✭✭

    My guess is that the data files aren't matching the mat44 input. I am reading the txt files as
    Vec4(column1, column2, column3, column4), Vec4(column5, column6, column7, column8), Vec4(column9, column10, column11, column12) and then add the final Vec4 as (0.0,0.0,0.0,1.0). That is what you wrote in a comment above.

    My suggestion therefor is to not use Mat44 data but Xfo data in the txt files, to avoid errors in the Xfo to Mat44 conversion (which I assume there are).

    Using the data set you linked above, I get this output, using all 300 text files as Mat44 inputs.

    I want it to be the same thing, but rather than Canvas actually reading each file when the timeline is moving, it should read all the files at the start and then just output the relevent array of Xfo's for each frame, putting the whole 2D array of all the arrays of Xfo's into a variable, so its stored.

    Either there is a logic glitch in your description or I am in Monday mode (i.e. I am stupid): When using the single frame node I wrote, you can simply change the xfos port to an IO port, so that you can add a variable array to "collect" the data over the timeline. However, this is not what you intend to do, as far as I understand.
    What you mean by "output the relevant array of Xfos for each frame" I do not get. So far I thought you wanted to have a SINGLE Xfo per frame, because the files you provided were frame-oriented, giving a SINGLE mat44 per file. So there is no "array of Xfos per frame", there is just a single Xfo per frame.

    Also, where is the 2d array? Both your sample graphs only use an array with one index (which correlates to the frame), so they are 1d (one dimensional).

    If you are sure that your mat44 data is correct (see above), my second guess is that it is not in the order Fabric expects (see above).

    I'd be happy to help, but I fear that I am misunderstanding something somewhere ...

    Marc

    all.PNG 12.2K

    Marc Albrecht - marc-albrecht.de - does things.

  • seltzdesignseltzdesign Posts: 80

    @malbrecht

    No problem, I dont think its Monday mode, its just hard to explain, that is all.

    So let me explain the text files. There is an animation happening, where objects are created (instances of a geometry). Each instance is generated from transforms which are expressed in a Mat44 matrix. At frame 0 there might be 1 instance (ie. 1 Mat44), but at frame 250 there are now 30 instances.

    Each file can contain any number of Mat44's and is represented by an array of Mat44's. Now I dont want to read each frame file in at runtime, but rather think its better to read all the text files when I set the folder and store their contents in a variable. Since there are many frames and each frame contains an array of Mat44's, we now have an array of arrays. Finally when I play the timeline (or anything else that creates the index) it gets the corresponding array of Mat44's from the array of arrays and outputs it. Now it doesnt need to actually read the file from the hard drive every time a certain frame is played.

    Actually the Xfo.SetFromMat44Array function works fine, so you can just leave them as Mat44 outputs. I was trying to use a more complicated method and only later discovered this node.

    I hope that clears it up.

    side note: I bet the word "array" hasnt been mentioned more in any thread yet :smiley:

  • malbrechtmalbrecht Fabric for Houdini Posts: 752 ✭✭✭

    Each file can contain any number of Mat44s

    Ah!

    That is not covered in the graph I uploaded. Monday mode after all.

    In that case you do want a "sort of 2d array", but your sample graphs don't cover that. If any frame can have any number of transforms, the array you get with index0=frame-number and index1=transform-id will have a variable size on its second index.
    Which is fine, no problem. Gimme a sec ...


    Marc Albrecht - marc-albrecht.de - does things.

  • malbrechtmalbrecht Fabric for Houdini Posts: 752 ✭✭✭

    OK, this is, obviously, not the final version, as the reader reads in ALL FRAMES on every frame. You will want to add a "is timeline == 0" and a variable to store the array of arrays in so that the cache is built only once.

    But I guess this is roughly what you were desiring?

    Marc


    Marc Albrecht - marc-albrecht.de - does things.

  • seltzdesignseltzdesign Posts: 80

    @malbrecht Absolutely right about that! It is what I want and now just need to solve the issue of only reading once. If I simply insert a cache node between your node and the Get[], its already doing it just fine and then plays back at a constant 60fps rather than around 12 without the caching.

    Strangely it is giving me those Math.Warning messages for every frame now (before it was just on certain frames).

  • malbrechtmalbrecht Fabric for Houdini Posts: 752 ✭✭✭

    Strangely it is giving me those Math.Warning messages for every frame now (before it was just on certain frames).

    Well, it's reading it all in at once, so everytime that node is called, the errors are thrown.
    "Cache" in Fabric is more like a "don't evaluate left from here" thing. If it's working with a cache, fine, else I'd use a variable. The attached version uses a variable and is throwing the warnings only once - on frame 1.

    Marc


    Marc Albrecht - marc-albrecht.de - does things.

2»
Sign In or Register to comment.