large KMZ asset decoding

[Jon Bradley]

Running into some big script timeouts here and wanted to know what
others have done to deal with it.

Loading a KMZ asset in is no problem and works fine. However, during
the decode process, because PV3D is utilizing the nochump ZIP
library, it gets stuck during the decode process quite often, even on
ZIP archives of around 1 MB in size.

Has anyone modified the Inflater.as file to add a timer to decode
large blocks instead of the whole thing at once (specifically the
decode function)?

thanks,

jon

[Matt Giger]

Jon,

I've never posted here before, but you've touched on something near
and dear to my heart, KML. I've implemented just what you are asking
for, but the work processing is tied in with my own game kernel so it
would be too hard to extricate from my code to integrate it with your
code.

Inflate.as was converted from the zlib source code at zlib-1.2.3/
contrib/puff/puff.c. My recommendation is to convert the C into
Actionscript and modify it along the way so that you can choose to
inflate each dynamic table based on how much time alloted is left.
Then upon re-entry of the inflation code, just pick up where you left
off. If you don't have a day to do that, you could try modifying
Inflate.as itself to do something similar, but I noticed that there
are some serious speed improvements that are included in puff.c.

I don't know what you are working on, but you might be interested in
using EarthBrowser to do what you need. Basically it's Google Earth
written in Flash. I'm set to release it shortly after I give a
presentation about it at Google next week.

If you want a sneak peek at the website plugin take a look at: http://www.earthbrowser.com/media/ebtest/
The AIR app is available at: http://www.earthbrowser.com

Good luck with your project.

Matt Giger

Post by Jon Bradley
Running into some big script timeouts here and wanted to know what
others have done to deal with it.
Loading a KMZ asset in is no problem and works fine. However, during
the decode process, because PV3D is utilizing the nochump ZIP
library, it gets stuck during the decode process quite often, even on
ZIP archives of around 1 MB in size.
Has anyone modified the Inflater.as file to add a timer to decode
large blocks instead of the whole thing at once (specifically the
decode function)?

[Ralph Hauwert]

That's incredibly good and well done. Applause!

ralph.

need

[Ralph Hauwert]

btw, it is a mod of papervision3D that is running on ?

Regards,
Ralph.

Post by Matt Giger
Jon,
I've never posted here before, but you've touched on something near
and dear to my heart, KML. I've implemented just what you are asking
for, but the work processing is tied in with my own game kernel so it
would be too hard to extricate from my code to integrate it with your
code.
Inflate.as was converted from the zlib source code at zlib-1.2.3/
contrib/puff/puff.c. My recommendation is to convert the C into
Actionscript and modify it along the way so that you can choose to
inflate each dynamic table based on how much time alloted is left.
Then upon re-entry of the inflation code, just pick up where you left
off. If you don't have a day to do that, you could try modifying
Inflate.as itself to do something similar, but I noticed that there
are some serious speed improvements that are included in puff.c.
I don't know what you are working on, but you might be interested in
using EarthBrowser to do what you need. Basically it's Google Earth
written in Flash. I'm set to release it shortly after I give a
presentation about it at Google next week.
If you want a sneak peek at the website plugin take a look at: http://www.earthbrowser.com/media/ebtest/
The AIR app is available at: http://www.earthbrowser.com
Good luck with your project.
Matt Giger

Post by Jon Bradley
Running into some big script timeouts here and wanted to know what
others have done to deal with it.
Loading a KMZ asset in is no problem and works fine. However, during
the decode process, because PV3D is utilizing the nochump ZIP
library, it gets stuck during the decode process quite often, even on
ZIP archives of around 1 MB in size.
Has anyone modified the Inflater.as file to add a timer to decode
large blocks instead of the whole thing at once (specifically the
decode function)?

_______________________________________________
Papervision3D mailing list
http://osflash.org/mailman/listinfo/papervision3d_osflash.org

[Jon Bradley]

Matt,

Post by Matt Giger
I don't know what you are working on, but you might be interested in
using EarthBrowser to do what you need. Basically it's Google Earth
written in Flash. I'm set to release it shortly after I give a
presentation about it at Google next week.

Dude, that's proper. Nice touch with the satellites ... :) Kudos.

I'll definitely check out the puff.c code and see if I can port it.
It would probably be better to do that than modify the nochump zip
stuff myself. Man, I wish defate/inflate on ByteArray were included
in the Player and not just AIR (and while I'm at it native decode/
encode of JPG and PNG ... ugh).

Post by Matt Giger
Good luck with your project.

And yours as well!

I'm actually just doing some testing work to see how feasible
something is right now.

I've got a whole separate project that I'm working on that retrofits
a rendering engine (raytracing/radiosity) into PV3D. Working out the
ZIP issues are going to be very useful for that from a data IO
standpoint.

thanks for the tips!

jon

[Ralph Hauwert]

Sorry for hijacking your thread like this Jon. I'm pretty sure that a
simple hack within the inflate loop of the zip libraries should do the
trick....I had a quick look. It's all about saving a state when your
timer hits, breaking the process, then coming back to it when needed.
I wish I had time to help you out with it, since it's an interesting
issue on itself, but currently I don't have enough to build a proper
solution for it. Maybe dave (nochump zip library) can help out ?

Regards,
Ralph.

Post by Jon Bradley
Running into some big script timeouts here and wanted to know what
others have done to deal with it.
Loading a KMZ asset in is no problem and works fine. However, during
the decode process, because PV3D is utilizing the nochump ZIP
library, it gets stuck during the decode process quite often, even on
ZIP archives of around 1 MB in size.
Has anyone modified the Inflater.as file to add a timer to decode
large blocks instead of the whole thing at once (specifically the
decode function)?
thanks,
jon
_______________________________________________
Papervision3D mailing list
http://osflash.org/mailman/listinfo/papervision3d_osflash.org

[Roy Wiggins]

While you're at it, I've had timeouts when loading collada files. Spreading
it out over multiple frames would be really useful.

Post by Ralph Hauwert
Sorry for hijacking your thread like this Jon. I'm pretty sure that a
simple hack within the inflate loop of the zip libraries should do the
trick....I had a quick look. It's all about saving a state when your
timer hits, breaking the process, then coming back to it when needed.
I wish I had time to help you out with it, since it's an interesting
issue on itself, but currently I don't have enough to build a proper
solution for it. Maybe dave (nochump zip library) can help out ?
Regards,
Ralph.

Post by Jon Bradley
Running into some big script timeouts here and wanted to know what
others have done to deal with it.
Loading a KMZ asset in is no problem and works fine. However, during
the decode process, because PV3D is utilizing the nochump ZIP
library, it gets stuck during the decode process quite often, even on
ZIP archives of around 1 MB in size.
Has anyone modified the Inflater.as file to add a timer to decode
large blocks instead of the whole thing at once (specifically the
decode function)?
thanks,
jon
_______________________________________________
Papervision3D mailing list
http://osflash.org/mailman/listinfo/papervision3d_osflash.org

_______________________________________________
Papervision3D mailing list
http://osflash.org/mailman/listinfo/papervision3d_osflash.org

[Jon Bradley]

Post by Roy Wiggins
While you're at it, I've had timeouts when loading collada files.
Spreading it out over multiple frames would be really useful.

For the actual parsing of the Collada files, I don't know if it's
fully implemented but it will asynchronously load the file and parse
if it you tell it to.

yourDaeReaderInstance = new DaeReader(true);

You'll may need to do either do one of the following (I don't know
when/where I got this info):

1. Modify org.ascollada.io.DaeReader to always use async (in the
constructor).

2. Change the loadDocument method in DaeReader as follows:

public function loadDocument( data:* ):DaeDocument
{
this.document = new DaeDocument( data, this.async );

_numAnimations = this.document.numQueuedAnimations;
_numGeometries = this.document.numQueuedGeometries;

if( _numGeometries > 0 )
{
readGeometries();
}
else
{
dispatchEvent( new Event(Event.COMPLETE) );
}

return this.document;
}

3. If you prefer to keep the async option as false by default, you'll
need to modify the Collada parsers themselves (ie,
org.papervision3d.objects.parsers.DAE) to make sure the DaeReader
instance (parser) is created with async, in the load method.

this.parser = new DaeReader(true);


There are a couple options with this.

cheers,

jon

[Jon Bradley]

Post by Ralph Hauwert
Sorry for hijacking your thread like this Jon. I'm pretty sure that a
simple hack within the inflate loop of the zip libraries should do the
trick....I had a quick look. It's all about saving a state when your
timer hits, breaking the process, then coming back to it when needed.

No worries... thanks for hopping in though and pointing out that
Matt's project rocked... he def. should get some attention on that
piece.

I've already done similar things to the coreilb JPG encoder and PNG
encoders, so I think doing it to the Inflate code won't be too much
of a concern. It looks like only two funcs need to be messed with
(and a couple vars with a timers) so I can't imagine it would take
that long. :)

Post by Ralph Hauwert
I wish I had time to help you out with it, since it's an interesting
issue on itself, but currently I don't have enough to build a proper
solution for it. Maybe dave (nochump zip library) can help out ?

Heh... thanks for at least looking though. Generally, I prefer not to
bug people to mod their code. It's probably best that I do it myself
and contribute the mod back. :)

cheers,

jon

[Jon Bradley]

Post by Ralph Hauwert
I'm pretty sure that a
simple hack within the inflate loop of the zip libraries should do the
trick....I had a quick look. It's all about saving a state when your
timer hits, breaking the process, then coming back to it when needed.
I wish I had time to help you out with it, since it's an interesting
issue on itself, but currently I don't have enough to build a proper
solution for it. Maybe dave (nochump zip library) can help out ?

Little update. I fixed up the classes yesterday and manage to get it
to load in chunks quite well. ZipFile and Inflate now extend
EventDispatcher and I've got a couple new event classes to handle
when a entry is parsed (passed as a property of the event) and an
error is thrown in the Inflate (passes the error). Also implemented
ProgressEvent dispatch.

Still looking to see if I can simplify the code a bit more or
streamline it.

Unpacking large ZIP archives is fine now. The next step is to modify
the KMZ parser to queue the decompression of the entries. As long as
the developer listens to the loaded event, all should work as it did
before.

- jb

[Drae]

Post by Jon Bradley
Little update. I fixed up the classes yesterday and manage to get it
to load in chunks quite well. ZipFile and Inflate now extend
EventDispatcher and I've got a couple new event classes to handle
when a entry is parsed (passed as a property of the event) and an
error is thrown in the Inflate (passes the error). Also implemented
ProgressEvent dispatch.
Still looking to see if I can simplify the code a bit more or
streamline it.
Unpacking large ZIP archives is fine now. The next step is to modify
the KMZ parser to queue the decompression of the entries. As long as
the developer listens to the loaded event, all should work as it did
before.

This was a LONG time ago, but I just ran into this issue myself, and I am
hoping that you are still around and willing share your code!

Thanks,
Drae


--
View this message in context: http://papervision3d.758870.n4.nabble.com/large-KMZ-asset-decoding-tp776366p3452340.html
Sent from the Papervision3D mailing list archive at Nabble.com.